{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.028107,
     "end_time": "2023-09-06T15:22:01.811311",
     "exception": false,
     "start_time": "2023-09-06T15:22:01.783204",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Titanic playground"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.024707,
     "end_time": "2023-09-06T15:22:01.861828",
     "exception": false,
     "start_time": "2023-09-06T15:22:01.837121",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Original Notebook: https://www.kaggle.com/code/marcpaulo/titanic-playground-for-new-kagglers-0-78\n",
    "\n",
    "The goal of this Notebook is to present a simple way to approach the **Titanic Competition** found in the **Kaggle** platform. The best model found here will be used in our Streamlit App ```titanic_streamlit/main_app/streamlit_app.py```\n",
    "\n",
    "The final solution (*Gradient Boosting Classifier*) presented here achieves a **0.78 score in the competition** (TOP 13~ on 07/09/2023). In the notebook ```titanic_streamlit/notebooks/training_grad_boost.ipynb``` you can find a short version of this *playground* notebook in which we train the Gradient Boosting Classifier directly.\n",
    "       \n",
    "The topics that we cover here are:\n",
    "          \n",
    "**1. Exploratory Data Analysis (EDA):**           \n",
    "    - we'll explore and analyze column by column     \n",
    "    - we'll see how we can detect missing values and outliers        \n",
    "    - we'll visualize each feature (column) to check their distribution and structure         \n",
    "    - we'll try to make an initial guess on the importance of each feature       \n",
    "    - we'll use the *Pandas* and *Seaborn* libraries           \n",
    "              \n",
    "**2. Data Preprocessing:**          \n",
    "    - we'll add a new feature (*feature extraction*)                            \n",
    "    - we'll see different ways to deal with missing values                        \n",
    "    - we'll use different modules from the *Sklearn* library to build clean and reusable *Pipelines* to process the data properly.\n",
    "              \n",
    "**3. Let's train some Models:**        \n",
    "    - we'll test different basic classification algorithms            \n",
    "    - we'll run many *GridSearch* with Cross-Validation for hyperparameter optimization      \n",
    "    - we'll plot the *GridSearch* results to compare the performance of all the different settings\n",
    "    \n",
    "**4. Save the final Model:**                \n",
    "    - train the best model architecture and save the trained parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.024931,
     "end_time": "2023-09-06T15:22:01.911814",
     "exception": false,
     "start_time": "2023-09-06T15:22:01.886883",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Let's get started!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enter your Project Path in which the 'titanic_streamlit' folder is located:\n",
    "\n",
    "notebook_config = {\n",
    "    'your_project_path': '/home/marc/Escritorio/titanic_streamlit',  # TODO: fill this!!!\n",
    "    \n",
    "    'save_model': True,\n",
    "    'model_file_name': 'trained_grad_boost.pkl',  # where the model is saved\n",
    "    'run_sanity_check': True  # try to load the model afer it's saved\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pickle\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.02521,
     "end_time": "2023-09-06T15:22:03.665607",
     "exception": false,
     "start_time": "2023-09-06T15:22:03.640397",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Read the train datasets using Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:03.718396Z",
     "iopub.status.busy": "2023-09-06T15:22:03.718010Z",
     "iopub.status.idle": "2023-09-06T15:22:03.757632Z",
     "shell.execute_reply": "2023-09-06T15:22:03.756241Z"
    },
    "papermill": {
     "duration": 0.069358,
     "end_time": "2023-09-06T15:22:03.760594",
     "exception": false,
     "start_time": "2023-09-06T15:22:03.691236",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_path = notebook_config['your_project_path'] + '/data/train.csv'\n",
    "df_train = pd.read_csv(data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:03.813339Z",
     "iopub.status.busy": "2023-09-06T15:22:03.812580Z",
     "iopub.status.idle": "2023-09-06T15:22:03.819693Z",
     "shell.execute_reply": "2023-09-06T15:22:03.818775Z"
    },
    "papermill": {
     "duration": 0.035887,
     "end_time": "2023-09-06T15:22:03.821973",
     "exception": false,
     "start_time": "2023-09-06T15:22:03.786086",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# lowercase all column names\n",
    "df_train.columns = [c.lower() for c in df_train.columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.024722,
     "end_time": "2023-09-06T15:22:03.872363",
     "exception": false,
     "start_time": "2023-09-06T15:22:03.847641",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "First, let's take a look at the structure of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:03.926067Z",
     "iopub.status.busy": "2023-09-06T15:22:03.924894Z",
     "iopub.status.idle": "2023-09-06T15:22:03.955257Z",
     "shell.execute_reply": "2023-09-06T15:22:03.953909Z"
    },
    "papermill": {
     "duration": 0.059881,
     "end_time": "2023-09-06T15:22:03.958238",
     "exception": false,
     "start_time": "2023-09-06T15:22:03.898357",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:04.011314Z",
     "iopub.status.busy": "2023-09-06T15:22:04.010681Z",
     "iopub.status.idle": "2023-09-06T15:22:04.016693Z",
     "shell.execute_reply": "2023-09-06T15:22:04.015515Z"
    },
    "papermill": {
     "duration": 0.035398,
     "end_time": "2023-09-06T15:22:04.019567",
     "exception": false,
     "start_time": "2023-09-06T15:22:03.984169",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(df_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.024953,
     "end_time": "2023-09-06T15:22:04.069737",
     "exception": false,
     "start_time": "2023-09-06T15:22:04.044784",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "- **THE TASK AT HAND:**\n",
    "    - a binary classification task. 11 features and 1 target ('Survived')\n",
    "    - 891 training samples\n",
    "            \n",
    "            \n",
    "- **COLUMNS DESCRIPTION**:   \n",
    "    - **passengerid**: passenger identifier\n",
    "    - **pclass**: ticket class {1: 1st, 2: 2nd, 3: 3rd}\n",
    "    - **name**: passenger's name and honorific title\n",
    "    - **sex**: 'male' or 'female'\n",
    "    - **age**: in years\n",
    "    - **sibsp**: number of siblings and spouses aboard\n",
    "    - **parch**: number of parents and children aboard\n",
    "    - **ticket**: ticket number\n",
    "    - **fare**: passenger fare\n",
    "    - **cabin**: cabin number\n",
    "    - **embarked**: port of embarkation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.025995,
     "end_time": "2023-09-06T15:22:04.121035",
     "exception": false,
     "start_time": "2023-09-06T15:22:04.095040",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 1. Exploratory Data Analysis (EDA)\n",
    "\n",
    "In this section, we are going to explore the data to summarize their main characteristics. We will use graphical tools to visualize the data and detect outliers, guess their distribution and try to gain a deeper understanding of the meaning of each feature (column)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:04.174244Z",
     "iopub.status.busy": "2023-09-06T15:22:04.173557Z",
     "iopub.status.idle": "2023-09-06T15:22:04.214473Z",
     "shell.execute_reply": "2023-09-06T15:22:04.212764Z"
    },
    "papermill": {
     "duration": 0.070856,
     "end_time": "2023-09-06T15:22:04.217145",
     "exception": false,
     "start_time": "2023-09-06T15:22:04.146289",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(df_train.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.02527,
     "end_time": "2023-09-06T15:22:04.268110",
     "exception": false,
     "start_time": "2023-09-06T15:22:04.242840",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**NULL VALUES:** (we will deal with them later)          \n",
    "    - **age**: 20% is missing        \n",
    "    - **cabin**: 77% is missing      \n",
    "    - **embarked**: 2 samples are missing        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.025322,
     "end_time": "2023-09-06T15:22:04.318983",
     "exception": false,
     "start_time": "2023-09-06T15:22:04.293661",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now, let's visit each column to check if there are outliers, and their relationship with the target ('survived')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.025018,
     "end_time": "2023-09-06T15:22:04.370101",
     "exception": false,
     "start_time": "2023-09-06T15:22:04.345083",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "* **passengerid**: it has zero information, so we can safely remove it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:04.423031Z",
     "iopub.status.busy": "2023-09-06T15:22:04.422609Z",
     "iopub.status.idle": "2023-09-06T15:22:04.431388Z",
     "shell.execute_reply": "2023-09-06T15:22:04.430121Z"
    },
    "papermill": {
     "duration": 0.038451,
     "end_time": "2023-09-06T15:22:04.433874",
     "exception": false,
     "start_time": "2023-09-06T15:22:04.395423",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_train = df_train.drop(columns=['passengerid'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.026465,
     "end_time": "2023-09-06T15:22:04.486135",
     "exception": false,
     "start_time": "2023-09-06T15:22:04.459670",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Before, exploring the other columns, let's first create a small function that will help us to visualize each feature and analyze their distribution and importance. This function is intended to be used with categorical features or numerical features with a small set of unique values (e.g. a finite set of natural numbers)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:04.542144Z",
     "iopub.status.busy": "2023-09-06T15:22:04.541761Z",
     "iopub.status.idle": "2023-09-06T15:22:04.553277Z",
     "shell.execute_reply": "2023-09-06T15:22:04.551903Z"
    },
    "papermill": {
     "duration": 0.042544,
     "end_time": "2023-09-06T15:22:04.555626",
     "exception": false,
     "start_time": "2023-09-06T15:22:04.513082",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def eda_bars_plot(df_: pd.DataFrame, \n",
    "                  feature: str,\n",
    "                  target: str = 'survived') -> None:\n",
    "    \"\"\"\n",
    "    Crates two bar plots:\n",
    "        - to check the distribution of 'feature'\n",
    "        - to check the relationship between the 'target'\n",
    "            and each category of 'feature'\n",
    "    \"\"\"\n",
    "\n",
    "    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5))\n",
    "    df_val_counts = df_[feature].value_counts() / len(df_)\n",
    "    df_val_counts = df_val_counts.reset_index().sort_values(by='index')\n",
    "    sns.barplot(x='index', y=feature, data=df_val_counts, ax=ax1)\n",
    "    ax1.set(xlabel=feature, \n",
    "            ylabel='% of each category', \n",
    "            title=feature + ': % value counts')\n",
    "\n",
    "    df_group_by = df_.groupby(feature)[target].mean().reset_index()\n",
    "    df_group_by = df_group_by.sort_values(by=feature)\n",
    "    sns.barplot(x=feature, y=target, data=df_group_by, ax=ax2)\n",
    "    ax2.set(xlabel=feature,\n",
    "            ylabel=f'% {target}', \n",
    "            title=f\"{feature}: prob of '{target}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.025285,
     "end_time": "2023-09-06T15:22:04.607397",
     "exception": false,
     "start_time": "2023-09-06T15:22:04.582112",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "* **pclass**: ticket class with values [1,2,3] = [Upper, Middle, Lower]        \n",
    "In the plots we can see that more than 50% of the passengers bought lower class tickets. Unsurprisingly: the higher your class, the more likely you are to survive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:04.660828Z",
     "iopub.status.busy": "2023-09-06T15:22:04.659796Z",
     "iopub.status.idle": "2023-09-06T15:22:05.220696Z",
     "shell.execute_reply": "2023-09-06T15:22:05.219304Z"
    },
    "papermill": {
     "duration": 0.59059,
     "end_time": "2023-09-06T15:22:05.223652",
     "exception": false,
     "start_time": "2023-09-06T15:22:04.633062",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "eda_bars_plot(df_=df_train, feature='pclass')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.025729,
     "end_time": "2023-09-06T15:22:05.276430",
     "exception": false,
     "start_time": "2023-09-06T15:22:05.250701",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "* **sex**: We can see that only 40% of the passengers were females, but women had more than a 70% chance of surviving!. This means that we could build a classifier that predicts survival only for women (and death for men), and would achieve a decent accuracy. It could serve as a good first result (baseline)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:05.331190Z",
     "iopub.status.busy": "2023-09-06T15:22:05.330513Z",
     "iopub.status.idle": "2023-09-06T15:22:05.851952Z",
     "shell.execute_reply": "2023-09-06T15:22:05.850929Z"
    },
    "papermill": {
     "duration": 0.551953,
     "end_time": "2023-09-06T15:22:05.854958",
     "exception": false,
     "start_time": "2023-09-06T15:22:05.303005",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "eda_bars_plot(df_=df_train, feature='sex')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.026245,
     "end_time": "2023-09-06T15:22:05.908246",
     "exception": false,
     "start_time": "2023-09-06T15:22:05.882001",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "* **age:** Almost 20% of the values are missing, but there don't seem to be any outliers since all values are within the normal range of ages.           \n",
    "          \n",
    "We definitely should NOT remove the rows with missing values since we would be losing a big portion of the data (too much information). Instead, we can *impute* the missing values. There are several ways to *impute* the missing values but, for simplicity, we will use the *median-imputer*. However, for more accurate results, other more advanced techniques may be used. In this case, for instance, we could use other categorical columns such as *'sex'*, *'pclass'* or the honorific title found in *'name'* to create a more specific *age-imputer* for each grup (category) of these features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:05.963204Z",
     "iopub.status.busy": "2023-09-06T15:22:05.962766Z",
     "iopub.status.idle": "2023-09-06T15:22:06.366702Z",
     "shell.execute_reply": "2023-09-06T15:22:06.365500Z"
    },
    "papermill": {
     "duration": 0.434774,
     "end_time": "2023-09-06T15:22:06.369479",
     "exception": false,
     "start_time": "2023-09-06T15:22:05.934705",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "ax_age = sns.histplot(x=df_train['age'])\n",
    "_ = ax_age.set(title=\"'age' column histogram\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.026214,
     "end_time": "2023-09-06T15:22:06.422476",
     "exception": false,
     "start_time": "2023-09-06T15:22:06.396262",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Looking at the plot below, it seems that the distribution of ages look quite similar for both the survivors and the non-survivors.          \n",
    "                \n",
    "**TODO:** In this section, we encourage you to explore the relationship between *'age'* and the other features. Maybe there is a combination of *'sex'+'age'* or *'pclass'+'sex'* groups which have something interesting to say in this classification problem (NOT TESTED!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:06.479449Z",
     "iopub.status.busy": "2023-09-06T15:22:06.479057Z",
     "iopub.status.idle": "2023-09-06T15:22:06.759542Z",
     "shell.execute_reply": "2023-09-06T15:22:06.757969Z"
    },
    "papermill": {
     "duration": 0.313384,
     "end_time": "2023-09-06T15:22:06.762862",
     "exception": false,
     "start_time": "2023-09-06T15:22:06.449478",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "sns.boxenplot(x=df_train['age'], y=df_train['survived'].astype('category'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.027742,
     "end_time": "2023-09-06T15:22:06.818686",
     "exception": false,
     "start_time": "2023-09-06T15:22:06.790944",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "* **name**: it has zero information, so we can safely remove it.\n",
    "\n",
    "**NOTE:** The honorific title (Mr, Mme, Lady, ...) can be extracted from the name and be used to create a new feature or to impute the missing *'age'* values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:06.895876Z",
     "iopub.status.busy": "2023-09-06T15:22:06.894409Z",
     "iopub.status.idle": "2023-09-06T15:22:06.905954Z",
     "shell.execute_reply": "2023-09-06T15:22:06.904942Z"
    },
    "papermill": {
     "duration": 0.055433,
     "end_time": "2023-09-06T15:22:06.909572",
     "exception": false,
     "start_time": "2023-09-06T15:22:06.854139",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_train = df_train.drop(columns=['name'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.027466,
     "end_time": "2023-09-06T15:22:06.968736",
     "exception": false,
     "start_time": "2023-09-06T15:22:06.941270",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "* **sibsp**: the number of siblings and spouses aboard. It looks ok: neither outliers nor missing values. \n",
    "        \n",
    "In the right-hand plot we can see that the you were more likely to survive if you had at list one sibling or a spouse abroad, but not too many of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:07.026986Z",
     "iopub.status.busy": "2023-09-06T15:22:07.025772Z",
     "iopub.status.idle": "2023-09-06T15:22:07.648162Z",
     "shell.execute_reply": "2023-09-06T15:22:07.646965Z"
    },
    "papermill": {
     "duration": 0.653731,
     "end_time": "2023-09-06T15:22:07.650672",
     "exception": false,
     "start_time": "2023-09-06T15:22:06.996941",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "eda_bars_plot(df_=df_train, feature='sibsp')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.028498,
     "end_time": "2023-09-06T15:22:07.707299",
     "exception": false,
     "start_time": "2023-09-06T15:22:07.678801",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "* **parch**: number of parents and children aboard. In the next section we will sum this feature and *'sibsp'* to create a new feature *'num_relatives'* (*feature extraction*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:07.765723Z",
     "iopub.status.busy": "2023-09-06T15:22:07.765312Z",
     "iopub.status.idle": "2023-09-06T15:22:08.394596Z",
     "shell.execute_reply": "2023-09-06T15:22:08.393376Z"
    },
    "papermill": {
     "duration": 0.661863,
     "end_time": "2023-09-06T15:22:08.397181",
     "exception": false,
     "start_time": "2023-09-06T15:22:07.735318",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "eda_bars_plot(df_=df_train, feature='parch')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.028422,
     "end_time": "2023-09-06T15:22:08.454629",
     "exception": false,
     "start_time": "2023-09-06T15:22:08.426207",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "- **Fare:** There is an extremely high value around 500. Since it is very distant from the rest of the values, we will regard it as an outlier. We will set the maximum value to 300 (so we will cut the greater values to this new maximum)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:08.513950Z",
     "iopub.status.busy": "2023-09-06T15:22:08.513476Z",
     "iopub.status.idle": "2023-09-06T15:22:09.036009Z",
     "shell.execute_reply": "2023-09-06T15:22:09.034804Z"
    },
    "papermill": {
     "duration": 0.555209,
     "end_time": "2023-09-06T15:22:09.038597",
     "exception": false,
     "start_time": "2023-09-06T15:22:08.483388",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "sns.histplot(x=df_train['fare'])\n",
    "plt.axvline(300, color='red', alpha=0.4, linestyle='--')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:09.099101Z",
     "iopub.status.busy": "2023-09-06T15:22:09.098690Z",
     "iopub.status.idle": "2023-09-06T15:22:09.104630Z",
     "shell.execute_reply": "2023-09-06T15:22:09.103471Z"
    },
    "papermill": {
     "duration": 0.039006,
     "end_time": "2023-09-06T15:22:09.107052",
     "exception": false,
     "start_time": "2023-09-06T15:22:09.068046",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# there are some outliers, lets cut the maximum value to be 300\n",
    "df_train.loc[df_train['fare'] > 300, 'fare'] = 300"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.029071,
     "end_time": "2023-09-06T15:22:09.165060",
     "exception": false,
     "start_time": "2023-09-06T15:22:09.135989",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "As we can see: the higher the fare, the more likely you are to survive.\n",
    "\n",
    "**TODO:** Our intuition tells us that the features *pclass* and *fare* might have some sort of correlation. Can you come up with a great plot to check if we are wrong? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:09.225326Z",
     "iopub.status.busy": "2023-09-06T15:22:09.224925Z",
     "iopub.status.idle": "2023-09-06T15:22:09.642779Z",
     "shell.execute_reply": "2023-09-06T15:22:09.641578Z"
    },
    "papermill": {
     "duration": 0.451217,
     "end_time": "2023-09-06T15:22:09.645444",
     "exception": false,
     "start_time": "2023-09-06T15:22:09.194227",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "sns.boxenplot(x=df_train['fare'], y=df_train['survived'].astype('category'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.029595,
     "end_time": "2023-09-06T15:22:09.705441",
     "exception": false,
     "start_time": "2023-09-06T15:22:09.675846",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "* **cabin**: let's remove this columns since there are too many null values (~80% of them are missing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:09.766803Z",
     "iopub.status.busy": "2023-09-06T15:22:09.766085Z",
     "iopub.status.idle": "2023-09-06T15:22:09.773383Z",
     "shell.execute_reply": "2023-09-06T15:22:09.772570Z"
    },
    "papermill": {
     "duration": 0.040909,
     "end_time": "2023-09-06T15:22:09.775881",
     "exception": false,
     "start_time": "2023-09-06T15:22:09.734972",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_train = df_train.drop(columns=['cabin'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.029051,
     "end_time": "2023-09-06T15:22:09.834778",
     "exception": false,
     "start_time": "2023-09-06T15:22:09.805727",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "* **embarked**: It turns out that the port of embarkation is more important than what you might expect. Let's see if our models are able to capture the importance of this feature!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:09.897659Z",
     "iopub.status.busy": "2023-09-06T15:22:09.897241Z",
     "iopub.status.idle": "2023-09-06T15:22:10.420568Z",
     "shell.execute_reply": "2023-09-06T15:22:10.419474Z"
    },
    "papermill": {
     "duration": 0.556878,
     "end_time": "2023-09-06T15:22:10.423279",
     "exception": false,
     "start_time": "2023-09-06T15:22:09.866401",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "eda_bars_plot(df_=df_train, feature='embarked')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.030291,
     "end_time": "2023-09-06T15:22:10.483650",
     "exception": false,
     "start_time": "2023-09-06T15:22:10.453359",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "* **ticket**: we will assume that it has zero information, so we will remove it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:10.547204Z",
     "iopub.status.busy": "2023-09-06T15:22:10.546184Z",
     "iopub.status.idle": "2023-09-06T15:22:10.553213Z",
     "shell.execute_reply": "2023-09-06T15:22:10.552139Z"
    },
    "papermill": {
     "duration": 0.041213,
     "end_time": "2023-09-06T15:22:10.555622",
     "exception": false,
     "start_time": "2023-09-06T15:22:10.514409",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_train = df_train.drop(columns=['ticket'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.02956,
     "end_time": "2023-09-06T15:22:10.615242",
     "exception": false,
     "start_time": "2023-09-06T15:22:10.585682",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 2. Data Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.02978,
     "end_time": "2023-09-06T15:22:10.676083",
     "exception": false,
     "start_time": "2023-09-06T15:22:10.646303",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this section, we will see different ways to deal with missing values depending on the type of features. We will also create a sequence of ordered *Sklearn Pipelines* to make our code reusable and avoid some typical problems such as *data leakage*. For the numerical features, we will apply a *MinMaxScaler* to map the values into the [0,1] interval. For the categorical features, we will apply a *OneHotEncoder*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.02986,
     "end_time": "2023-09-06T15:22:10.737062",
     "exception": false,
     "start_time": "2023-09-06T15:22:10.707202",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "* **Feature extraction:** let's create a new feature from the already-existing ones. The number of relatives (family size) might be useful here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:10.800363Z",
     "iopub.status.busy": "2023-09-06T15:22:10.799720Z",
     "iopub.status.idle": "2023-09-06T15:22:10.806259Z",
     "shell.execute_reply": "2023-09-06T15:22:10.805341Z"
    },
    "papermill": {
     "duration": 0.041071,
     "end_time": "2023-09-06T15:22:10.808625",
     "exception": false,
     "start_time": "2023-09-06T15:22:10.767554",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "## a new feature: num_relatives = sibsp + parch\n",
    "df_train['num_relatives'] = df_train['sibsp'] + df_train['parch']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.031985,
     "end_time": "2023-09-06T15:22:10.871514",
     "exception": false,
     "start_time": "2023-09-06T15:22:10.839529",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "It tourns out that 60% of the passengers were alone (at least with no family aboard). Small families (1,2,3 people) seem to be more likely to survive (more than 50% chance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:10.933728Z",
     "iopub.status.busy": "2023-09-06T15:22:10.932985Z",
     "iopub.status.idle": "2023-09-06T15:22:11.609655Z",
     "shell.execute_reply": "2023-09-06T15:22:11.608073Z"
    },
    "papermill": {
     "duration": 0.711127,
     "end_time": "2023-09-06T15:22:11.612792",
     "exception": false,
     "start_time": "2023-09-06T15:22:10.901665",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "eda_bars_plot(df_=df_train, feature='num_relatives')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.031443,
     "end_time": "2023-09-06T15:22:11.675411",
     "exception": false,
     "start_time": "2023-09-06T15:22:11.643968",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's separate the training features (*df_train*) from the target (*y_train*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:11.740194Z",
     "iopub.status.busy": "2023-09-06T15:22:11.739765Z",
     "iopub.status.idle": "2023-09-06T15:22:11.746951Z",
     "shell.execute_reply": "2023-09-06T15:22:11.745637Z"
    },
    "papermill": {
     "duration": 0.042404,
     "end_time": "2023-09-06T15:22:11.749460",
     "exception": false,
     "start_time": "2023-09-06T15:22:11.707056",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "y_train = df_train['survived'].values  # [0,1]\n",
    "df_train = df_train.drop(columns=['survived'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:11.813269Z",
     "iopub.status.busy": "2023-09-06T15:22:11.812564Z",
     "iopub.status.idle": "2023-09-06T15:22:11.826410Z",
     "shell.execute_reply": "2023-09-06T15:22:11.825309Z"
    },
    "papermill": {
     "duration": 0.048587,
     "end_time": "2023-09-06T15:22:11.828816",
     "exception": false,
     "start_time": "2023-09-06T15:22:11.780229",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.032875,
     "end_time": "2023-09-06T15:22:11.894315",
     "exception": false,
     "start_time": "2023-09-06T15:22:11.861440",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now, we will implement the *Preprocessing Pipeline* using *Sklearn*. Remember that columns *'age', 'fare', and 'embarked'* have at least one missing value. The transformations we will apply are:            \n",
    "    - **pclass**: OneHotEncoder  (values [1,2,3])             \n",
    "    - **sex**: OneHotEncoder  (values [0,1])            \n",
    "    - **age**: Median-Imputer + MinMaxScaler            \n",
    "    - **sibsp**: MinMaxScaler           \n",
    "    - **parch**: MinMaxScaler           \n",
    "    - **fare**: MinMaxScaler           \n",
    "    - **embarked**: MostFrequentImputer + OneHotEncoder  (values ['S','C','Q'])        \n",
    "    - **num_relatives**: MinMaxScaler       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:11.958953Z",
     "iopub.status.busy": "2023-09-06T15:22:11.958193Z",
     "iopub.status.idle": "2023-09-06T15:22:12.540436Z",
     "shell.execute_reply": "2023-09-06T15:22:12.539316Z"
    },
    "papermill": {
     "duration": 0.617988,
     "end_time": "2023-09-06T15:22:12.543166",
     "exception": false,
     "start_time": "2023-09-06T15:22:11.925178",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler, StandardScaler, OneHotEncoder\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:12.606826Z",
     "iopub.status.busy": "2023-09-06T15:22:12.606413Z",
     "iopub.status.idle": "2023-09-06T15:22:12.616362Z",
     "shell.execute_reply": "2023-09-06T15:22:12.615059Z"
    },
    "papermill": {
     "duration": 0.044581,
     "end_time": "2023-09-06T15:22:12.618946",
     "exception": false,
     "start_time": "2023-09-06T15:22:12.574365",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# First, create three small pipelines to stack more than one Transformer\n",
    "# for a specific feature. Those features that only require one \n",
    "# transformation are handled by the final 'preprocessor' here below.\n",
    "age_pipe = Pipeline(steps=[\n",
    "    ('age_imp', SimpleImputer(strategy='median')),\n",
    "    ('age_scale', MinMaxScaler())\n",
    "])\n",
    "embarked_pipe = Pipeline(steps=[\n",
    "    ('embarked_imp', SimpleImputer(strategy='most_frequent')),\n",
    "    ('embarked_onehot', OneHotEncoder(drop=None))\n",
    "])\n",
    "\n",
    "# Let's create the final 'preprocessor'\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        ('age_pipe', age_pipe, ['age']),\n",
    "        ('embarked_pipe', embarked_pipe, ['embarked']),\n",
    "        ('minmax_scaler', MinMaxScaler(), ['fare', 'sibsp', 'parch', 'num_relatives']),\n",
    "        ('pclass_onehot', OneHotEncoder(drop=None), ['pclass']),\n",
    "        ('sex_onehot', OneHotEncoder(drop='first'), ['sex'])\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.032134,
     "end_time": "2023-09-06T15:22:12.683070",
     "exception": false,
     "start_time": "2023-09-06T15:22:12.650936",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 3. Let's train some Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.031275,
     "end_time": "2023-09-06T15:22:12.745976",
     "exception": false,
     "start_time": "2023-09-06T15:22:12.714701",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Once the data is ready, we can move on to the **'Modelling'** section. We will test different basic Machine Learning models and perform **Grid-Search Cross-Validation** to optimize the most interesting hyperparameters. In order to visualize the CV results, we will plot them using different techniques depending on the number of hyperparameters to optimize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:12.810179Z",
     "iopub.status.busy": "2023-09-06T15:22:12.809702Z",
     "iopub.status.idle": "2023-09-06T15:22:12.815395Z",
     "shell.execute_reply": "2023-09-06T15:22:12.813970Z"
    },
    "papermill": {
     "duration": 0.041018,
     "end_time": "2023-09-06T15:22:12.818011",
     "exception": false,
     "start_time": "2023-09-06T15:22:12.776993",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.032855,
     "end_time": "2023-09-06T15:22:12.882859",
     "exception": false,
     "start_time": "2023-09-06T15:22:12.850004",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's start with one of the simplest Classification models: **LogisticRegression**. We will run a Grid Search to optimize the regularization parameter **'C'**. Then, we will plot the results to visualize how the accuracy depends on the 'C' value, and select the best model (the one achieving the best validation accuracy in the cross-validation).\n",
    "        \n",
    "In the plot below, we can see that the best result of **~79% accuracy** is achieved with **'C'= 1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:12.946823Z",
     "iopub.status.busy": "2023-09-06T15:22:12.946024Z",
     "iopub.status.idle": "2023-09-06T15:22:16.281784Z",
     "shell.execute_reply": "2023-09-06T15:22:16.280365Z"
    },
    "papermill": {
     "duration": 3.371016,
     "end_time": "2023-09-06T15:22:16.284732",
     "exception": false,
     "start_time": "2023-09-06T15:22:12.913716",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "log_reg = Pipeline(steps=[\n",
    "    ('preprocessor', preprocessor),\n",
    "    ('log_reg', LogisticRegression())\n",
    "])\n",
    "log_reg_params = {'log_reg__C': [1e-3, 1e-2, 1e-1, 1, 1e1, 1e2, 1e3]}\n",
    "\n",
    "log_reg_cv = GridSearchCV(estimator=log_reg, \n",
    "                          param_grid=log_reg_params,\n",
    "                          scoring='accuracy',\n",
    "                          n_jobs=-1,\n",
    "                          cv=5,\n",
    "                          verbose=1,\n",
    "                          return_train_score=True\n",
    "                         )\n",
    "log_reg_cv.fit(df_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:16.351608Z",
     "iopub.status.busy": "2023-09-06T15:22:16.351208Z",
     "iopub.status.idle": "2023-09-06T15:22:16.356904Z",
     "shell.execute_reply": "2023-09-06T15:22:16.355785Z"
    },
    "papermill": {
     "duration": 0.043553,
     "end_time": "2023-09-06T15:22:16.360583",
     "exception": false,
     "start_time": "2023-09-06T15:22:16.317030",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "print('best params:', log_reg_cv.best_params_)\n",
    "print('best validation score:', log_reg_cv.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:16.427529Z",
     "iopub.status.busy": "2023-09-06T15:22:16.426624Z",
     "iopub.status.idle": "2023-09-06T15:22:16.778443Z",
     "shell.execute_reply": "2023-09-06T15:22:16.777010Z"
    },
    "papermill": {
     "duration": 0.389465,
     "end_time": "2023-09-06T15:22:16.782224",
     "exception": false,
     "start_time": "2023-09-06T15:22:16.392759",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "plt.plot(\n",
    "    range(len(log_reg_params['log_reg__C'])), \n",
    "    log_reg_cv.cv_results_['mean_train_score'],\n",
    "    label='train',\n",
    "    color='blue'\n",
    ")\n",
    "plt.plot(\n",
    "    range(len(log_reg_params['log_reg__C'])),\n",
    "    log_reg_cv.cv_results_['mean_test_score'],\n",
    "    label='test',\n",
    "    color='green'\n",
    ")\n",
    "plt.xticks(ticks=range(len(log_reg_params['log_reg__C'])),\n",
    "           labels=log_reg_params['log_reg__C'])\n",
    "plt.legend()\n",
    "plt.axvline(x=3, linestyle='--', color='red', alpha=0.3)\n",
    "plt.xlabel(\"'C' vals\")\n",
    "plt.ylabel('accuracy')\n",
    "plt.title(\"LogisticRegression GridSearchCV for 'C' param\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.042493,
     "end_time": "2023-09-06T15:22:16.867721",
     "exception": false,
     "start_time": "2023-09-06T15:22:16.825228",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now, we will test the **Support Vector Classifier** with the **Radial Kernel**. Here the goal is to optimize the regularization parameter **'C'**, and the kernel coefficient **'gamma'**. Since we have two hyperparameters to optimize, here we will display a *heat map* to show the results on how the accuracy depends on 'C' and 'gamma'.\n",
    "        \n",
    "In the heat map below, we can see that the best result of **~81% accuracy** is achieved with **'C'= 1** and **'gamma' = 1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:16.945653Z",
     "iopub.status.busy": "2023-09-06T15:22:16.944960Z",
     "iopub.status.idle": "2023-09-06T15:22:24.476735Z",
     "shell.execute_reply": "2023-09-06T15:22:24.475277Z"
    },
    "papermill": {
     "duration": 7.574157,
     "end_time": "2023-09-06T15:22:24.479821",
     "exception": false,
     "start_time": "2023-09-06T15:22:16.905664",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "svc = Pipeline(steps=[\n",
    "    ('preprocessor', preprocessor),\n",
    "    ('svc', SVC(kernel='rbf'))\n",
    "])\n",
    "svc_params = {\n",
    "    'svc__C': [1e-3, 1e-2, 1e-1, 1, 1e1, 1e2, 1e3],\n",
    "    'svc__gamma': [1e-2, 1e-1, 1, 10, 100]\n",
    "}\n",
    "              \n",
    "svc_cv = GridSearchCV(\n",
    "    estimator=svc, \n",
    "    param_grid=svc_params,\n",
    "    scoring='accuracy',\n",
    "    n_jobs=-1,\n",
    "    cv=5,\n",
    "    verbose=1,\n",
    "    return_train_score=False\n",
    ")\n",
    "svc_cv.fit(df_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:24.550973Z",
     "iopub.status.busy": "2023-09-06T15:22:24.550512Z",
     "iopub.status.idle": "2023-09-06T15:22:24.556765Z",
     "shell.execute_reply": "2023-09-06T15:22:24.555488Z"
    },
    "papermill": {
     "duration": 0.044986,
     "end_time": "2023-09-06T15:22:24.559633",
     "exception": false,
     "start_time": "2023-09-06T15:22:24.514647",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "print('best params:', svc_cv.best_params_)\n",
    "print('best validation score:', svc_cv.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:24.630692Z",
     "iopub.status.busy": "2023-09-06T15:22:24.630288Z",
     "iopub.status.idle": "2023-09-06T15:22:25.147962Z",
     "shell.execute_reply": "2023-09-06T15:22:25.146561Z"
    },
    "papermill": {
     "duration": 0.556028,
     "end_time": "2023-09-06T15:22:25.150725",
     "exception": false,
     "start_time": "2023-09-06T15:22:24.594697",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "ss = svc_cv.cv_results_['mean_test_score'].reshape(\n",
    "    (len(svc_params['svc__C']), len(svc_params['svc__gamma']))\n",
    ")\n",
    "ax = sns.heatmap(ss, vmin=0.6, vmax=0.85, linewidth=0.5, annot=True)\n",
    "_ = ax.set(\n",
    "    xlabel='gamma', \n",
    "    ylabel='C', \n",
    "    title='SVC Validation Accuracy GridSearchCV',\n",
    "    xticklabels=svc_params['svc__gamma'],\n",
    "    yticklabels=svc_params['svc__C']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.035577,
     "end_time": "2023-09-06T15:22:25.221268",
     "exception": false,
     "start_time": "2023-09-06T15:22:25.185691",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now, we will try the **Random Forest Classifier**. The hyperparameters of this model are the number of decision trees **'n_estimators'**, and the maximum depth of each tree **'max_depth'**. Since we have two hyperparameters to optimize, here we will display a *heat map* to show the results on how the accuracy depends on 'n_estimators' and 'max_depth'. In general, the *Random Forest* algorithm usually works well with the default values for the rest of hyperparameters.\n",
    "\n",
    "Note that the Random Forest can learn from features in different scales because they are processed separately. Although we could simplify our *'preprocessor'* by removing the *MinMaxScalers*, we will keep it the same for all models.\n",
    "        \n",
    "In the heat map below, we can see that the best result of **~83% accuracy** is achieved with **'n_estimators'= 100** and **'max_depth' = 8**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:22:25.294353Z",
     "iopub.status.busy": "2023-09-06T15:22:25.293922Z",
     "iopub.status.idle": "2023-09-06T15:23:36.476785Z",
     "shell.execute_reply": "2023-09-06T15:23:36.475277Z"
    },
    "papermill": {
     "duration": 71.260843,
     "end_time": "2023-09-06T15:23:36.517475",
     "exception": false,
     "start_time": "2023-09-06T15:22:25.256632",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "rf = Pipeline(steps=[\n",
    "    ('preprocessor', preprocessor),\n",
    "    ('rf', RandomForestClassifier(random_state=12345))\n",
    "])\n",
    "rf_params = {\n",
    "    'rf__n_estimators': [50, 100, 200, 400, 600, 800, 1000],\n",
    "    'rf__max_depth': [1, 4, 8, 10]\n",
    "}\n",
    "\n",
    "rf_cv = GridSearchCV(\n",
    "    estimator=rf, \n",
    "    param_grid=rf_params,\n",
    "    scoring='accuracy',\n",
    "    n_jobs=-1,                  \n",
    "    cv=5,          \n",
    "    verbose=1,\n",
    "    return_train_score=True\n",
    ")\n",
    "rf_cv.fit(df_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:23:36.591215Z",
     "iopub.status.busy": "2023-09-06T15:23:36.590814Z",
     "iopub.status.idle": "2023-09-06T15:23:36.596336Z",
     "shell.execute_reply": "2023-09-06T15:23:36.595415Z"
    },
    "papermill": {
     "duration": 0.045353,
     "end_time": "2023-09-06T15:23:36.598935",
     "exception": false,
     "start_time": "2023-09-06T15:23:36.553582",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "print('best RandomForest params:', rf_cv.best_params_)\n",
    "print('best RandomForest cv accuracy:', rf_cv.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:23:36.672843Z",
     "iopub.status.busy": "2023-09-06T15:23:36.671655Z",
     "iopub.status.idle": "2023-09-06T15:23:37.195886Z",
     "shell.execute_reply": "2023-09-06T15:23:37.194218Z"
    },
    "papermill": {
     "duration": 0.563961,
     "end_time": "2023-09-06T15:23:37.198610",
     "exception": false,
     "start_time": "2023-09-06T15:23:36.634649",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "ss = rf_cv.cv_results_['mean_test_score'].reshape(\n",
    "    (len(rf_params['rf__max_depth']), len(rf_params['rf__n_estimators']))\n",
    ")\n",
    "ax = sns.heatmap(ss, vmin=0.7, vmax=0.85, linewidth=0.5, annot=True)\n",
    "_ = ax.set(\n",
    "    xlabel='n_estimators', \n",
    "    ylabel='max_depth', \n",
    "    title='RandomForest Accuracy GridSearchCV',\n",
    "    xticklabels=rf_params['rf__n_estimators'],\n",
    "    yticklabels=rf_params['rf__max_depth']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.036417,
     "end_time": "2023-09-06T15:23:37.271616",
     "exception": false,
     "start_time": "2023-09-06T15:23:37.235199",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The next algorithm is the **Gradient Boosting Classifier**. This algorithm performs a gradient optimization on a *loss function*. With this kind of *additive* model (such as the *Random Forest*), the default hyperparameter values tend to perform well enough. Here we will optimize the **'n_estimators'** (number of decision trees), and the **'learning_rate'**. As we did with other models, we will show the results in a *heat map*.\n",
    "\n",
    "In the heat map below, we can see that the best result of **~83% accuracy** is achieved with **'n_estimators'= 400** and **'learning_rate' = 0.1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:23:37.348438Z",
     "iopub.status.busy": "2023-09-06T15:23:37.348024Z",
     "iopub.status.idle": "2023-09-06T15:24:18.815558Z",
     "shell.execute_reply": "2023-09-06T15:24:18.814404Z"
    },
    "papermill": {
     "duration": 41.54614,
     "end_time": "2023-09-06T15:24:18.855692",
     "exception": false,
     "start_time": "2023-09-06T15:23:37.309552",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "\n",
    "grad_boost = Pipeline(steps=[\n",
    "    ('preprocessor', preprocessor),\n",
    "    ('grad_boost', GradientBoostingClassifier(random_state=12345))\n",
    "])\n",
    "grad_boost_params = {\n",
    "    'grad_boost__n_estimators': [50, 100, 200, 400, 600, 800, 1000],\n",
    "    'grad_boost__learning_rate': [1e-4, 1e-3, 1e-2, 1e-1]}\n",
    "\n",
    "grad_boost_cv = GridSearchCV(\n",
    "    estimator=grad_boost, \n",
    "    param_grid=grad_boost_params,\n",
    "    scoring='accuracy',\n",
    "    n_jobs=-1,\n",
    "    cv=5,\n",
    "    verbose=1,\n",
    "    return_train_score=False\n",
    ")\n",
    "grad_boost_cv.fit(df_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:24:18.931853Z",
     "iopub.status.busy": "2023-09-06T15:24:18.931385Z",
     "iopub.status.idle": "2023-09-06T15:24:18.937567Z",
     "shell.execute_reply": "2023-09-06T15:24:18.936380Z"
    },
    "papermill": {
     "duration": 0.0475,
     "end_time": "2023-09-06T15:24:18.940194",
     "exception": false,
     "start_time": "2023-09-06T15:24:18.892694",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "print('best GradientBoosting params:', grad_boost_cv.best_params_)\n",
    "print('best GradientBoosting cv accuracy:', grad_boost_cv.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:24:19.016651Z",
     "iopub.status.busy": "2023-09-06T15:24:19.016230Z",
     "iopub.status.idle": "2023-09-06T15:24:19.493326Z",
     "shell.execute_reply": "2023-09-06T15:24:19.491872Z"
    },
    "papermill": {
     "duration": 0.518537,
     "end_time": "2023-09-06T15:24:19.495877",
     "exception": false,
     "start_time": "2023-09-06T15:24:18.977340",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "ss = grad_boost_cv.cv_results_['mean_test_score'].reshape(\n",
    "    (len(grad_boost_params['grad_boost__learning_rate']), \n",
    "     len(grad_boost_params['grad_boost__n_estimators']))\n",
    ")\n",
    "ax = sns.heatmap(ss, vmin=0.6, vmax=0.85, linewidth=0.5, annot=True)\n",
    "_ = ax.set(\n",
    "    xlabel='n_estimators', \n",
    "    ylabel='learning_rate', \n",
    "    title='GradientBoosting Accuracy GridSearchCV',\n",
    "    xticklabels=grad_boost_params['grad_boost__n_estimators'],\n",
    "    yticklabels=grad_boost_params['grad_boost__learning_rate']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.039877,
     "end_time": "2023-09-06T15:24:19.574709",
     "exception": false,
     "start_time": "2023-09-06T15:24:19.534832",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The last algorithm is the **K-Nearest Neighbors Classifier**. We will use the *Euclidean* distance (the metric by default) and we will play with the **'n_neighbors** hyperparam.\n",
    "\n",
    "In the plot below, we can see that the best result of **~80% accuracy** is achieved with **'n_neighbors'= 32**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:24:19.653984Z",
     "iopub.status.busy": "2023-09-06T15:24:19.653544Z",
     "iopub.status.idle": "2023-09-06T15:24:21.744777Z",
     "shell.execute_reply": "2023-09-06T15:24:21.743856Z"
    },
    "papermill": {
     "duration": 2.133863,
     "end_time": "2023-09-06T15:24:21.747533",
     "exception": false,
     "start_time": "2023-09-06T15:24:19.613670",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "knn = Pipeline(steps=[\n",
    "    ('preprocessor', preprocessor),\n",
    "    ('knn', KNeighborsClassifier())\n",
    "])\n",
    "knn_params = {\n",
    "    'knn__n_neighbors': [2, 4, 8, 16, 32, 64, 92, 128]\n",
    "}\n",
    "\n",
    "knn_cv = GridSearchCV(estimator=knn, \n",
    "                     param_grid=knn_params,\n",
    "                     scoring='accuracy',\n",
    "                      n_jobs=-1,\n",
    "                      cv=5,\n",
    "                      verbose=1,\n",
    "                      return_train_score=True\n",
    "                     )\n",
    "knn_cv.fit(df_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:24:21.827723Z",
     "iopub.status.busy": "2023-09-06T15:24:21.827332Z",
     "iopub.status.idle": "2023-09-06T15:24:21.833203Z",
     "shell.execute_reply": "2023-09-06T15:24:21.832069Z"
    },
    "papermill": {
     "duration": 0.049865,
     "end_time": "2023-09-06T15:24:21.836601",
     "exception": false,
     "start_time": "2023-09-06T15:24:21.786736",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "print('best KNearestNeighbors params:', knn_cv.best_params_)\n",
    "print('best KNearestNeighbors cv accuracy:', knn_cv.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:24:21.916407Z",
     "iopub.status.busy": "2023-09-06T15:24:21.916015Z",
     "iopub.status.idle": "2023-09-06T15:24:22.244284Z",
     "shell.execute_reply": "2023-09-06T15:24:22.242802Z"
    },
    "papermill": {
     "duration": 0.372118,
     "end_time": "2023-09-06T15:24:22.247311",
     "exception": false,
     "start_time": "2023-09-06T15:24:21.875193",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "plt.plot(\n",
    "    range(len(knn_params['knn__n_neighbors'])), \n",
    "    knn_cv.cv_results_['mean_train_score'],\n",
    "    label='train',\n",
    "    color='blue'\n",
    ")\n",
    "plt.plot(\n",
    "    range(len(knn_params['knn__n_neighbors'])),\n",
    "    knn_cv.cv_results_['mean_test_score'],\n",
    "    label='test',\n",
    "    color='green'\n",
    ")\n",
    "plt.xticks(ticks=range(len(knn_params['knn__n_neighbors'])),\n",
    "           labels=knn_params['knn__n_neighbors'])\n",
    "plt.legend()\n",
    "plt.axvline(x=4, linestyle='--', color='red', alpha=0.3)\n",
    "plt.xlabel(\"'n_neighbors' vals\")\n",
    "plt.ylabel('accuracy')\n",
    "plt.title(\"KNearestNeighbour GridSearchCV for 'n_neighbors' param\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.039027,
     "end_time": "2023-09-06T15:24:22.325888",
     "exception": false,
     "start_time": "2023-09-06T15:24:22.286861",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**TODO:** Try other classifiers! Some of the most promising algorithms are:   \n",
    "    * **CatBoost**  ---  [link](https://catboost.ai/)                \n",
    "    * **XGBoost**  ---  [link](https://xgboost.readthedocs.io/en/stable/python/python_api.html)            \n",
    "    * **LightGBM**  ---  [link](https://lightgbm.readthedocs.io/en/stable/Python-API.html)\n",
    "       \n",
    "You can also explore the **Deep Learning** univers and build a Feed-Forward Neural Network using [Sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html), [TensorFlow](https://www.tensorflow.org/api_docs/python/tf), or [PyTorch](https://pytorch.org/docs/stable/index.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Save the final model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have seen in the previous section, it turns out that the **Gradient Boosting** with 400 estimators and a learning rate of 0.1 is the best model we came up with (the best cross-validation accuracy). However, looking at the heatmap of the Cross-Validation results, we can see that with 100 estimators we also get an ~83% validation accuracy, with a 75% of reduction in the number of estimators. It means that both values of 'n_estimator' perform similarly.\n",
    "\n",
    "For the sake of simplicity, let's choose the *'small'* model of **100 Decision Trees** and a **learning rate of 0.1**. A bigger model is more likely to overfit. Usually in Machine Learning: the simpler, the better.\n",
    "\n",
    "Now, we will run a *10-Fold Cross-Validation* again to double-check the performance of the final model (expected to be an **83% accuracy**).\n",
    "           \n",
    "The last step is to train the final model using the whole dataset and save the trained parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-09-06T15:24:22.569934Z",
     "iopub.status.busy": "2023-09-06T15:24:22.569214Z",
     "iopub.status.idle": "2023-09-06T15:24:23.427613Z",
     "shell.execute_reply": "2023-09-06T15:24:23.426347Z"
    },
    "papermill": {
     "duration": 0.904594,
     "end_time": "2023-09-06T15:24:23.430690",
     "exception": false,
     "start_time": "2023-09-06T15:24:22.526096",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "best_grad_boost = Pipeline(steps=[\n",
    "    ('preprocessor', preprocessor),\n",
    "    ('grad_boost', GradientBoostingClassifier(\n",
    "        n_estimators=100, \n",
    "        learning_rate=0.1, \n",
    "        random_state=12345\n",
    "    ))\n",
    "])\n",
    "\n",
    "best_grad_boost_acc = cross_val_score(\n",
    "    estimator=best_grad_boost,\n",
    "    X=df_train,\n",
    "    y=y_train,\n",
    "    scoring='accuracy',\n",
    "    cv=10,\n",
    "    n_jobs=-1\n",
    ")\n",
    "\n",
    "print('best GradientBoosting acc (mean) =', round(np.mean(best_grad_boost_acc), 2))\n",
    "print('best GradientBoosting acc (std)  =', round(np.std(best_grad_boost_acc), 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grad_boost = Pipeline(steps=[\n",
    "    ('preprocessor', preprocessor),\n",
    "    ('grad_boost', GradientBoostingClassifier(\n",
    "        n_estimators=100,\n",
    "        learning_rate=0.1,\n",
    "        random_state=12345\n",
    "    ))\n",
    "])\n",
    "\n",
    "grad_boost.fit(df_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the trained model\n",
    "\n",
    "save_model_path = (\n",
    "        notebook_config['your_project_path'] + \n",
    "        '/models/' + \n",
    "        notebook_config['model_file_name']\n",
    "    )\n",
    "\n",
    "if notebook_config['save_model']:\n",
    "    \n",
    "    \n",
    "    with open(save_model_path, 'wb') as out_file:\n",
    "        pickle.dump(grad_boost, out_file)\n",
    "    print(f\"Grad Boost model saved in:\\n'{save_model_path}'\")\n",
    "\n",
    "else:\n",
    "\n",
    "    print('According to the notebook_config, the model is NOT saved')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SANITY CHECK: Load the model\n",
    "\n",
    "if notebook_config['run_sanity_check']:\n",
    "    with open(save_model_path, 'rb') as in_file:\n",
    "        loaded_model = pickle.load(in_file)\n",
    "\n",
    "    print(f\"Grad Boost model loaded from:\\n'{save_model_path}'\")\n",
    "    print('train score:', loaded_model.score(df_train, y_train))\n",
    "\n",
    "else:\n",
    "    \n",
    "    print('According to the notebook_config, do NOT run Sanity Check')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 157.935357,
   "end_time": "2023-09-06T15:24:26.682445",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2023-09-06T15:21:48.747088",
   "version": "2.4.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
